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Job  Knowledge,  R&tee  Familiarity, 

Conceptual  Similarity,  and  Halo  Error: 

An  Exploration 

Halo  error  (Thorndike,  1920),  or  illusory  correlation 
(Bingham,  1939),  has  been  regarded  as  a  particularly 
pervasive  form  of  inadvertent  rater  judgment  bias  (Borman, 
1977;  Cooper,  1981b).  Early  investigators  of  trait  and 
performance  Judgments  discovered  that  rating  dimension 
intercorrelations  tended  to  be  higher  than  the  presumed 
independence  of  the  dimensions  led  them  to  expect  (cf. 
Rugg,  1922;  Thorndike,  1920;  Wells,  1907).  Bingham  (1939) 
distinguished  between  •’true"  and  "illusory"  halo,  noting 
that  jobs  require  the  use  of  shared  abilities  across 
performance  dimensions.  Thus,  some  dimensional 
intercorrelation  is  inevitable,  with  the  degree  of  halo 
error  hinging  on  the  extent  to  which  rating 
Intercorrelations  exceed  the  corresponding  true  score 
intercorrelations . 

Conceptualizations  of  the  nature  of  halo  error  have 
been  remarkably  consistent.  Thorndike  characterized  halo 
as  the  tendency  by  a  rater  to  "think  of  the  person  in 
general  as  rather  good  or  rather  inferior  and  to  color  the 
judgment  of  the  separate  qualities  by  this  general 
feeling"  (1920,  p.  25).  It  was  argued  that  the  recall  of 
information  for  ratings  would  be  affected  by  this  general 
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impression.  Raters  would  tend  to  overestimate  the  ratee 
on  other  dimensions  seen  as  consistent  with  the  general 
impression,  thereby  inflating  the  intercorrelations. 
Newcomb  (1931)  discovered  that  retrospective  ratings  of 
personality  traits  were  more  highly  intercorrelated  than 
direct  observational  evidence  provided  by  the  same  raters 
indicated  was  warranted.  Newcomb  suggested  that  the 
inflated  intercorrelations  were  due  to  the  raters' 
implicit  covariance  assumptions,  that  is,  the  "logical 
presuppositions  in  the  minds  of  the  raters  rather  than 
from  actual  behavior"  <1931,  pi  288).  These  two 
perspectives  of  halo  differ  somewhat  on  the  nature  of  the 
hypothesized  source  of  bias,  but  imply  a  similar  process. 

The  conventional  view  has  held  that  halo  is 
particularly  sensitive  to  the  raters'  familiarity  or 
knowledge  of  the  ratee  and  the  ratee' s  job,  with  halo  more 
likely  under  conditions  in  which  raters  lack 
performance-relevant  knowledge.  The  arguments  and 
evidence  for  this  assertion  are  largely  a  deduction  from 
measurement  theory.  For  example,  a  number  of  studies  have 
reported  positive  relationships  between  rater  knowledge  of 
the  ratee  and  rating  scale  reliability  and  validity  (cf. 
Ferguson,  1949;  Hollander,  1954,  1957;  Kornhauser,  1926). 
The  assumption  has  been  that  improvements  in  reliability 
and  validity  associated  with  increased  rater  knowledge  are 
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also  associated  with  corresponding  reductions  in  all  forms 
of  error  bias  including  halo.  There  have  been  few  direct 
empirical  tests  which  actually  evaluated  this  hypothesis 
vis  a  vis  halo.  Studies  by  Brown  (1968)  and  Koltuv 
(1962),  however,  reported  that  rating  dimension 
intercorrelations  were  lower  for  raters  indicating  greater 
familiarity  with  the  ratee. 

At  a  general  level,  the  process  implications  have 
been  that  global  impressions  or  pre-existing  cognitive 
schemata  are  more  likely  to  be  sources  of  performance 
judgments  when  the  rater  lacks  performance-relevant 
information  (Borman,  1974;  Cooper,  1981a,  1981b;  DeCotiis, 
1977;  Freeberg,  1969;  Thorndike  &  Hagen,  1961;  Wherry, 
1952) . 

Cooper  (1981a,  1981b)  has  offered  a  more  richly 
articulated  process  hypothesis.  It  draws  upon  the  work  of 
Shweder  (1975,  1980)  and  D' Andrade  (1974)  who  asserted 
that  the  conceptual  similarities  among  rating  dimension  • 
labels  were  the  source  of  implicit  rater  covariance 
schemata.  In  other  words,  raters  err  by  mistaking 
semantic  category  resemblance  for  behavioral  covariance 
(Tversky  &  Kahneman,  1974).  By  equating  this  view  with 
Newcomb's  (1931)  suggestion  that  inflated  rating 
intercorrelations  resulted  from  raters'  implicit 
conceptual  schemes.  Cooper  (1981b)  proposed  conceptual 
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similarity  as  a  prevalent  source  of  halo  error  in 
performance  ratings.  This  is  thought  especially  to  be  the 
case  when  ratings  are  obtained  under  “difficult  memory 
conditions"  (Shweder,  1980,  p.  75) --where  raters  lack  job 
knowledge,  knowledge  of  the  ratee,  and/or  opportunities 
for  information  decay  increase.  Judgments  made  during 
recall  will  be  systematically  biased  in  the  direction  of 
the  pre-existing  conceptual  scheme,  resulting  in  halo 
error.  It  should  be  noted  that  Cooper  (1981a)  has 
interpreted  this  process  to  mean  that  ratings  on 
conceptually  similar  categories  will  covary  at  a  higher 
level  than  ratings  on  dimensions  that  are  perceived  as 
less  similar,  not  that  conceptually  similar  dimensions 
will  receive  the  same  rating  (cf.  Lamiell,  Foss,  & 

Cavenee,  1980;  Shweder,  1980). 

Some  support  for  what  has  been  termed  the  'systematic 
distortion  hypothesis '  has  been  provided  by  a  series  of 
studies.  Using  ratings  of  personality  traits,  these 
studies  have  shown  that  raters'  conceptual  similarity 
judgments  were  more  strongly  associated  with  rating  and 
true  score  covariance,  than  rating  covariance  was  with 
true  score  covariance  (cf.  D' Andrade  &  Shweder,  1980; 
Shweder,  1975).  Thus,  raters  were  concluded  to  be  more 
sensitive  to  conceptual  similarity  than  to  actual 
behavioral  covariation  in  making  their  judgments.  Berman 
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&  Kenny  (1976)  also  reported  that  trait  similarity 
judgments  provided  by  raters  were  associated  with  rating 
intercorrelations . 

Cooper  (1981a)  applied  these  notions  to  the  area  of 
performance  appraisal,  exploring  the  relationship  of  blind 
raters'  conceptual  similarity  covariance  judgments  with 
behavior-based  performance  ratings  and  true  scores.  He 
reported  that  the  conceptual  similarity  judgments  were 
reasonable  predictors  of  previously  observed  rating 
intercorelations  for  three  different  jobs  (r's  =  .92,  .57, 
and  .57).  However,  when  compared  with  rating  and  true 
score  intercorrelations  obtained  from  Borman  (1977),  the 
hypothesis  was  not  supported.  Rating  covariance  was  more 
strongly  associated  with  true  score  covariance  (£  *  .89) 
than  with  conceptual  similarity  judgments  (£  *  .55).  As 
Cooper  (1981a)  correctly  noted,  the  study  did  not 
adequately  address  the  systematic  distortion  hypothesis 
because  there  was  no  difficult  memory  condition.  Ratings 
had  been  obtained  immediately  after  raters  viewed  a 
videotaped  performance  segment.  Thus,  the  raters  were  all 
equally  knowledgeable  concerning  the  ratee's  job  and  level 
of  performance,  and  no  opportunity  for  memory  decay  was 
apparent.  Moreover,  the  relationship  of  conceptual 
similarity  to  halo  error  was  never  demonstrated. 

In  addition  to  this  operational  problem,  there  was  an 
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additional  limitation.  The  Achilles  Heel  of  performance 
rating  research  has  been  the  difficulty  of  generating 
estimates  of  true  scores  against  which  ratings  can  be 
evaluated.  The  videotape  stimuli  methodology  developed  by 
Borman  (1977)  was  a  major  advance  in  this  respect. 

However,  this  method  does  possess  a  number  of  limitations 
that  make  its  applicability  problematic  with  respect  to 
the  systematic  distortion  hypothesis.  For  example,  it  is 
questionable  whether  the  raters,  typically  college 
students,  are  knowledgeable  about  the  job  performance  they 
attempt  to  characterize.  The  performance  sequences  they 
observe  are  generally  very  brief  in  duration  (5  to  11 
minutes),  as  opposed  to  more  realistic  observational 
periods  spanning  several  months.  Their  attention  is 
highly  focused  during  the  observation  sequence  both  by  the 
experimental  task  demands,  as  well  as  by  the  selective 
nature  of  the  camera  as  observer — unlike  more  realistic 
observational  situations  where  many  stimuli  compete  for 
attention.  Thus,  in  its  present  state  of  development, 
this  method  appears  to  lack  ecological  validity. 

In  order  to  address  these  limitations,  this  study 
used  baseball  players  as  target  ratees.  This  provided  a 
naturally  occurring  performance  domain  which  unfolded  over 
a  lengthy  time  period  (approximately  6  months)  and  allowed 
for  different  levels  of  rater  job  knowledge  and  knowledge 


Job  Knowledge 
9 


of  the  ratee,  thus  more  adequately  simulating  a  realistic 
rating  task.  More  importantly,  however,  the  performance 
domain  provided  objective  measures  which  could  be  employed 
as  true  score  estimates.  Although  such  measures  cannot  be 
regarded  as  true  scores  in  the  strict  psychometric  sense 
because  some  error  of  measurement  is  present,  they  are 
superior  to  the  true  scores  derived  from  the  videotape 
methodology  for  this  particular  study.  The  true  score 
estimates  used  in  the  videotape  method  are  derived  from 
pooled  rater  judgments.  Thus,  if  conceptual  similarity, 
or  some  other  cognitive  scheme,  is  consistently  used  by 
raters,  that  scheme  will  be  reflected  in  the  pooled 
judgments  and  the  true  score  estimates.  The  use  of 
independent  measures  would  eliminate  this  potential 
confound. 

The  present  study  represents  an  effort  to  reexamine 
the  applicability  of  the  systematic  distortion  hypothesis 
to  the  domain  of  performance  ratings  and  its  effect  on 
rater  halo.  Specifically,  conceptual  similarity  judgments 
are  hypothesized  to  be  more  strongly  associated  with 
rating  covariance  and  rating  covariance  less  strongly 
associated  with  true  score  covariance  when  raters  lack  job 
and/or  ratee  knowledge.  In  addition,  it  is  anticipated 
that  halo  will  be  greater  under  these  low  knowledge 
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Method 

Subjects 

Raters  were  drawn  from  the  introductory  psychology 
courses  of  a  large  mid-western  university.  Their 
participation  was  voluntary  and  they  received  a  nominal 
amount  of  credit  toward  their  grade.  Altogether,  192 
students  participated  in  the  research  with  186  providing 
useful  responses. 

Procedure 

Performance  dimensions.  The  categories  of  baseball 
player  performance  were  limited  to  measures  of  offensive 
behavior.  The  seven  dimensions  that  were  assessed 
included  the  following:  batting  average,  home  runs, 
extra-base-hits,  runs  scored,  strike  outs,  stolen  bases, 
and  runs  createdl.  Objective  measures  of  these  indices 
for  the  1983  baseball  season  were  obtained  directly  from 
the  official  major  league  sources  for  the  ratees  in  the 
study 2. 

Conceptual  similarity  judgments.  In  order  to  assess 
the  raters'  pre-existing  conceptual  similarity  schemes, 
subjects  were  asked  to  judge  the  degree  of  similarity  for 
all  the  possible  pairings  of  the  seven  performance 
dimensions.  Instructions  were  provided  which  explained 
the  concept  of  similarity  and  provided  examples.  In 
addition,  definitions  of  the  performance  dimensions  were 
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also  included  to  ensure  that  all  raters  had  a  common  basis 
for  the  judgements.  The  ratings  were  made  using  a  scale 
ranging  from  0  to  100  percent  similar.  This  procedure 
resulted  in  21  conceptual  similarity  ratings  C7  (7  -  1)  / 

2  3. 

Knowledge  conditions.  The  systematic  distortion 
hypothesis  specifically  states  that  there  must  be  a  delay 
between  behavioral  observation  and  recall  for  the 
conceptual  similarity  scheme  to  bias  the  recalled 
observations.  This  requirement  was  built  into  the  rating 
task  by  gathering  the  ratings  at  the  end  of  the  1983 
baseball  season.  Thus,  the  performance  ratings  were  based 
on  behavior  accumulated  over  a  six  month  period.  It  was 
hypothesized  that  the  conceptual  scheme  would  exhibit  a 
greater  association  with  rating  covariation  where  the 
rater  had  poor  job  knowledge  and/or  lacked 
performance-relevant  ratee  information.  These  two 
knowledge  conditions  were  established  in  the  following 
manner. 

First,  a  job  knowledge  condition  was  created  by 
having  the  raters  indicate  the  extent  to  which  they  were 
active  observers  of  the  sport.  Ratings  were  made  on  a 
seven-point  scale  with  the  following  anchors:  l=not  very 
knowledgeable- -I  seldom  or  never  read  about  or  watch 
baseball;  4*somewhat  knowledgeable--!  sometimes  follow  the 
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sport  in  newspapers  and  watch  it  on  TV;  and,  7=extremely 
knowledgeable- -I  follow  the  sport  in  the  newspaper  and 
watch  it  on  TV  regularly.  The  job  knowledge  responses 
were  normally  distributed  and  ranged  over  the  entire 
scale.  The  job  knowledge  condition  was  created  by 
dichotomizing  responses  on  the  rating  of  baseball 
knowledge,  defining  responses  four  and  below  as  low  job 
knowledge  (n=90)  and  those  greater  than  four  as  high  job 
knowledge  ( n=96 ) . 

Second,  a  ratee  familiarity  condition  was 
established.  This  condition  was  difficult  to  define  a 
priori.  Therefore,  raters  were  provided  with  a  list  of  30 
major  league  players /ratees  selected  to  cover  a  range  of 
possible  rater  exposure  to  the  players.  Raters  were 
instructed  to  select  two  players  from  the  list:  one  known 
very  well  and  one  whom  they  knew  little  about.  The  wide 
range  of  visibility  of  the  players /ratees  was  to  ensure, 
that  low  job  knowledge  raters  could  identify  ratees  they 
knew  about  and  high  job  knowledge  raters  could  find  ratees 
they  knew  little  about.  As  a  validity  check,  raters  also 
indicated  the  extent  of  their  familiarity  with  each  ratee 
they  selected  on  a  seven  point  scale.  The  familiarity 
rating  difference  between  well  known  and  little  known 
ratees  was  significant  <£,  =  18.56,  p  <  .001),  thereby 
verifying  the  manipulation. 
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Ratings  and  true  scores.  Raters  selected  their 
ratees  as  instructed  and  provided  ratings  on  the  seven 
performance  dimensions  using  seven-point  rating  scales. 
The  appropriate  ratee's  objective  performance  measures 
were  attached  to  each  set  of  ratings. 

Results 

The  design  for  the  study  incorporated  two  knowledge 
factors,  job  and  ratee,  each  with  two  levels.  To  address 
the  systematic  distortion  hypothesis,  intercorrelation 
matrices  for  the  ratings  and  true  scores  were  generated 
for  the  four  cells.  One-half  of  each  of  the  matrices, 
excluding  the  diagonal,  was  transformed  to  z  scores 
(Edwards,  1954)  and  arrayed  as  vectors  corresponding  to 
the  conceptual  similarity  judgments.  Intermatrix 
correlations  were  computed  among  the  Intercorrelation 
vectors  and  raters'  mean  conceptual  similarity  judgments. 
The  Intermatrix  correlations  are  displayed  in  Table  1. 


Insert  Table  1  about  here 


The  results  indicated  that  the  conceptual 
similarity-rating  covariance  relationship  was 
significantly  different  for  the  high  and  low  job  knowledge 
conditions  (j  *  1.93,  p  <  .05,  one-tailed,  Cohen  &  Cohen, 
1983)  when  rating  a  well-known  ratee,  as  predicted.  High 
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job  knowledge  racers'  conceptual  similarity  schemes  showed 
little  association  with  rating  covariance  (r  =  .21),  while 
low  job  knowledge  raters'  conceptual  similarity  schemes 
were  strongly  associated  with  rating  covariance  (£  =  .72). 
Moreover,  the  rating-true  score  covariance  association  was 
higher  than  the  conceptual  similarity-rating  covariance 
association  for  high  job  knowledge  raters,  while  the 
reverse  was  true  for  the  low  job  knowledge  condition. 

Looking  across  the  ratee  knowledge  condition,  there 
was  a  small  but  nonsignificant  decline  in  the  conceptual 
similarity-rating  covariance  association  for  low  job 
knowledge  raters.  Thus,  it  would  appear  that  raters 
lacking  job  knowledge  tended  to  rely  on  their  pre-existing 
conceptual  schemes  in  making  ratings,  regardless  of  how 
well  they  knew  the  ratee.  There  was,  however,  a 
substantial  effect  for  high  job  knowledge  raters.  The 
degree  of  conceptual  similarity-rating  covariance 
association  jumped  from  £  =  .21  to  £  =  .82  (t  *  3.20,  p  < 
.01,  one-tailed;  see  Cohen  &  Cohen,  1983)  when  they  rated 
players  not  well-known  to  them.  Moreover,  that 
association  exceeded  the  rating-true  score  covariance 
relationship  as  predicted  by  the  systematic  distortion 
hypothesis.  These  findings  replicate  the  results  reported 
by  Shweder  (1975)  and  indicate  that  pre-existing 
conceptual  similarity  schemes  are  strongly  associated  with 
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rating  covariance  when  raters  lack  job  knowledge  and/or 
are  unfamiliar  with  the  ratee's  performance.  It  should  be 
noted,  however,  that  the  results  do  not  clearly  indicate 
that  conceptual  similarity  serves  as  a  template  for  the 
observed  rating  covariance. 

Conventional  wisdom  holds  that  halo  will  be  greater 
under  low  job  or  ratee  knowledge  conditions.  To  evaluate 
the  effects  of  job  and  ratee  knowledge  on  halo,  ratings 
were  standardized  for  the  total  sample  and  the  raters' 
standard  deviation  across  the  rating  dimensions  was 
computed  as  an  index  of  rating  halo.  Ratings  were 
standardized  in  order  to  eliminate  the  effect  of  rating 
dimension  mean  differences  on  the  halo  index  (Pulakos  & 
Schmitt,  1984).  Means  and  standard  deviations  of  the  halo 
measure  are  displayed  in  Table  2. 


Insert  Table  2  about  here 


The  measure  was  subjected  to  a  repeated  measures 
analysis  of  variance.  Homogeneity  of  variance  assumptions 
were  met  (Box  M  =  6.83,  X2  *  6.75,  n.s.).  Results  of  the 
analysis  revealed  significant  main  effects  for  job 
knowledge  (£  (1,84)  *  7.34,  p  <  .05),  and  ratee  knowledge 
(F  (1,84)  *  4.40,  p  <  .05),  while  the  interaction  was  not 
significant  (F  (1,84)  »  .13,  n.s.).  The  effect  is  clearly 
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indicated  by  the  means  reported  in  Table  2.  Halo  is 
greater  under  both  low  knowledge  conditions,  as  predicted. 

Although  the  results  link  halo  to  situations  in  which 
raters  lack  performance-relevant  job  and  ratee 
information,  the  specific  role  of  conceptual  similarity  as 
the  major  source  of  halo  (cf.  Cooper,  1981a)  remains  to  be 
demonstrated. 

Finally,  in  order  to  address  the  conventional  wisdom 
concerning  the  relationship  of  job  and  ratee  knowledge  and 
rating  reliability  and  validity,  reliability  and  validity 
analyses  were  undertaken.  The  determination  of  rating 
reliability  for  all  the  cells  in  the  design  was 
problematic  because  ratees  were  not  common  across  raters. 
However,  average  reliabilities  collapsed  over  knowledge  of 
baseball  indicated  greater  interrater  agreement  for 
well-known  ratees  <£  »  .96)  than  for  little  known  ratees 
(r  *  .62),  as  would  be  expected.  Moreover,  raters  who 
reported  greater  knowledge  of  baseball  were  somewhat  more 
reliable  (r  ■  .93)  than  less  knowledgeable  raters  (r  «* 

.81)  when  rating  a  well-known  ratee.  Other  estimates  were 
impossible  to  ascertain. 

Validity  coefficients  (correlations  of  the  ratings 
with  the  objective  performance  measures)  are  displayed  in 
Table  3. 
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Insert  Table  3  about  here 


These  results  clearly  indicate  that  job  knowledge  was 
an  important  factor  for  valid  ratings.  Raters  with  low 
knowledge  of  baseball  were  unable  to  predict  performance 
regardless  of  how  familiar  they  were  with  the  ratee.  The 
results  for  high  job  knowledge  raters  show  that  ratee 
familiarity  also  affected  the  validity  of  the  ratings. 
Ratings  were  better  predictors  when  well-known  players 
were  rated. 

Discussion 

The  results  of  this  study  provide  support  for  the 
notion  that  conceptual  similarity  is  more  strongly 
associated  with  rating  covariation  under  conditions  of  low 
job  and  ratee  performance-relevant  information.  It  also 
supports  the  commonly  held  belief  that 

performance-relevant  knowledge  and  halo  negatively  covary. 
Cooper's  (1981a)  contention  that  conceptual  similarity  is 
a  source  of  halo  was  not  contradicted,  although  no  direct 
support  was  evident.  These  results  reiterate  the  current 
focus  on  rater  job  knowledge  and  ratee  observation 
(familiarity)  as  features  critical  to  understanding  the 
rating  process. 

A  second  important  outcome  of  this  research  was  the 
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success  of  using  sports  figures,  where  objective 
performance  measures  are  available,  as  target  stimuli  in 
rating  research.  This  method  has  the  potential  to  provide 
realistic  performance  rating  simulations  without  many  of 
the  limitations  associated  with  the  current  videotape 
methodology.  It  is  hoped  that  this  study  will  stimulate 
further  research  of  this  nature. 

Finally,  there  were  several  issues  which  arose  in 
this  research  which  should  be  addressed.  Although  raters 
provided  their  own  conceptual  similarity  schemes,  unlike 
previous  research  which  used  blind  raters  to  provide  such 
ratings  (cf.  Cooper,  1981a;  Shweder,  1975),  the 
relationship  of  conceptual  similarity  to  rating  covariance 
was  limited  to  a  group  level  analysis.  That  is,  it  was 
not  possible  to  assess  the  effect  of  individual- level 
conceptual  similarity  on  individual  rating  covariance, 
since  raters  rated  only  one  player  under  each  ratee 
knowledge  condition.  Thus,  it  is  possible  that  the 
group-level  analysis  overstates  the  extent  to  which 
conceptual  similarity  is  associated  with  individual -level 
rating  covariance  (James,  1982).  Having  only  one  rating 
per  subject  in  a  given  knowledge  condition  also  precluded 
an  opportunity  to  determine  whether  individual- level 
conceptual  similarity  was  indeed  a  source  of 
individual -level  illusory  halo  as  suggested  by  Cooper 
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(1981a).  This  could  be  rectified  in  future  research  by 
having  all  raters  rate  a  larger  number  (e.g.,  n=10)  of  the 
same  ratees.  It  would  then  be  possible  to  directly 
estimate  illusory  halo  at  the  level  of  the  individual 
rater  by  comparing  the  degree  of  rating  intercorrelation 
with  the  intercorrelation  of  the  true  score  estimates,  and 
to  explore  the  relationship  of  conceptual  similarity  or 
some  other  cognitive  organizational  scheme  to  illusory 
halo. 


v.v:M  i 
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Footnotes 

lRuns  Created  (RC)  is  an  index  of  the  player's 
ability  to  increase  his  team's  score  through  all  the 
elements  of  his  batting  record.  It  was  computed  by: 

(Hits  +  Walks)  (Total  Bases) 

RC  =  - - -  ; 


(At  bats  +  Walks) 


see  James  (1982). 


2The  official  sources  were:  Sports  Information 


Center,  Boston,  MA  (American  League)  and  Elias  Sports 
Bureau,  New  York,  NY  (National  League). 
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